[1] 0 1 1 1 1 0 1 0 1 0 1 0 0 1 0 0 1 0 1 1
Main results
PSCI 2270 - Week 10
Department of Political Science, Vanderbilt University
November 2, 2023
Project Updates
Experiments and Their Types
Example: Censorship in China
Example: Conjoint and Audit
What does “\(T_i\) causes \(Y_i\)” mean? \(\Rightarrow\) counterfactuals or what-if’s
Two potential outcomes:
Causal effect: \(Y_i (1) − Y_i (0)\)
Fundamental problem of causal inference: Only one of the two potential outcomes is observable
What we can estimate instead: \[\text{Difference-in-means} = \frac{1}{n_T} \sum_{i = 1}^{n_T} Y_i (1) − \frac{1}{n_C} \sum_{i = 1}^{n_C} Y_i (0)\]
Key idea: Randomization of the treatment makes the treatment and control groups “identical” on average
The two groups are expected to be similar in terms of all characteristics (both observed and unobserved)
In the gay marriage example: Send canvassers to knock on people doors and randomly assign some households to canvassers who are members of LGBT community
A reproducible procedure that generates assignments with known probability between \(0\) and \(1\)
The treatment is the variable that the researcher manipulates in an effort to study its effects on outcomes
For example:
The outcome variable (also known as the dependent variable) is a quantity that may be influenced by a treatment
For example:
Random assignment of subjects to treatments
Non-interference (no spillovers): Subject’s potential outcomes reflect only whether they receive the treatment themselves
Excludability: Subject’s potential outcomes respond only to the defined treatment, not other extraneous factors that may be correlated with treatment
Random assignment: Researchers selected the units by looking at their pre-treatment characteristics \(\Rightarrow\) Fail!
Non-interference: Study participants are neighbors and those who were contacted by LGBT community members talked to those who were not \(\Rightarrow\) Fail!
Excludability: LGBT community members who conducted interviews knew about study purposes and tried to make respondents more receptive to the survey \(\Rightarrow\) Fail!
What if your treatment is too abstract and is not represented by concrete intervention?
Solution: A manipulation check is an attempt to verify that the treatment received was akin to the treatment that the researcher intended to deploy
What if participants don’t receive the intended treatment?
Consider an extreme case in which no one in the treatment group receives the treatment
What if study participants guess what you are studying?
Lab experiments: Participants are invited into lab (in the university or in the field) to participate in the study
Survey experiments: During the survey (online/phone/in-person) some respondents are assigned to receive certain questions/vignettes or the order of questions is randomized; the outcomes are measured during the same survey
Field experiments: Researchers administer intervention in naturalistic and unobtrusive setting and measure outcomes later via surveys or administrative data
What is the research hypothesis?
What are the experimental conditions?
Who (or what) are the subjects?
How are the subjects assigned to treatment(s)?
In what context does the experiment take place?
How are outcomes measured?
How do researchers estimate average treatment effect?
Are there any threats to inference? Think about random assignment, non-interference, excludability, experimenter effects, treatment meaning
Summary:
Why do autocrats want to do censorship?
How can this be achieved with censorship?
Hypothesis: Authors claim that the collective action theory is true but not the theory about positive image of the government.
What is the treatment in the study? Two separate treatments
Who (or what) are the subjects?
Design: Factorial audit experiment
How did they assign the treatment?
What is the context of the study?
Possible issues?
What are the outcomes they measure?
How do they analyze the data?
Should we be concerned about non-interference? Likely No!
Should we be concerned about experimenter effects? Yes!
Any other concerns?
Summary:
There is a perception that Democrats and Republicans vary in terms of their preferences on immigration policy
Alternative theories
What is the treatment in the study? Many separate treatments
Who (or what) are the subjects?
Design: Conjoint experiment
How did they assign the treatment?
What is the context of the study?
Possible issues?
What are the outcomes they measure?
How do they analyze the data?
Should we be concerned about experimenter effects? Yes!
Should we be concerned about non-interference? Yes!
Should we be concerned about many tests that they run? Yes!
Any other concerns?